State-of-the-art database systems are required to store and process massive amounts of data with extremely high efficiency. For example, a database storage solution for Internet business advertising accounts may require sorting, filtering, and paginating hundreds of millions of data records in sub-second time.
Current techniques for implementing very large databases include using federation schemes, wherein multiple databases are linked to a common central interface. In a federated database system, data is horizontally partitioned across multiple component databases, and federation keys are assigned to map data queries to corresponding component databases. While federation schemes are scalable to achieve greater capacity, they lack the flexibility and speed to dynamically adjust database access based on current network load. Furthermore, the assignment of related data rows to a single federation atomic unit may limit the amount of data that can be accommodated.
Accordingly, it would be desirable to provide a novel low-latency query processor capable of processing queries for arbitrary amounts of data, featuring dynamic adjustment and optimization depending on network load.